{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "82e45532",
   "metadata": {},
   "source": [
    "# OpenAI Chat with Temperature\n",
    "\n",
    "Temperature in deep learning is a parameter usually used to adjust the probability distribution of the predicted outputs. It is also known as softmax temperature or softmax scaling. In simple terms, it controls the level of confidence that a neural network has in its predictions. It helps to **increase the diversity of the model's outputs**.\n",
    "\n",
    "For temperature in OpenAI chat request, \"higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic\" as explained in [OpenAI documentation](https://platform.openai.com/docs/api-reference/chat/create).\n",
    "\n",
    "GPTCache also enables a similar `temperature` parameter at request, in range of [0.0, 2.0], which works at two stages:\n",
    "- control the possibility of sending request to OpenAI directly without searching in cache\n",
    "- affect the selection of final answer from potential answers retrieved from cache\n",
    "\n",
    "Let us try the adapted OpenAI Chat API with GPTCache enabled and see how `temperature` affects output given the same question. \n",
    "\n",
    "\n",
    "## Set Cache\n",
    "\n",
    "Initiate GPTCache with preferred configurations and modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c6e46737",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "from gptcache import cache, Config\n",
    "from gptcache.manager import manager_factory\n",
    "from gptcache.embedding import Onnx\n",
    "from gptcache.processor.post import temperature_softmax\n",
    "from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation\n",
    "from gptcache.adapter import openai\n",
    "\n",
    "\n",
    "onnx = Onnx()\n",
    "data_manager = manager_factory(\"sqlite,faiss\", vector_params={\"dimension\": onnx.dimension})\n",
    "\n",
    "cache.init(\n",
    "    embedding_func=onnx.to_embeddings,\n",
    "    data_manager=data_manager,\n",
    "    similarity_evaluation=SearchDistanceEvaluation(),\n",
    "    post_process_messages_func=temperature_softmax\n",
    "    )\n",
    "# cache.config = Config(similarity_threshold=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a47ceff8",
   "metadata": {},
   "source": [
    "## Getting Started"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "15c6dfee",
   "metadata": {},
   "outputs": [],
   "source": [
    "cache.set_openai_key()\n",
    "question = 'what is github'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7517ddd",
   "metadata": {},
   "source": [
    "### default: temperature=0.0\n",
    "\n",
    "If temperature is not specified in request, it will use the default value 0. When temperature is 0, it will firstly search through cache and return the most confident answer retrieved from cache. If there is no satisfactory answer available in cache, it will continue on sending request to OpenAI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "04683730",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time elapsed: 7.906\n",
      "GitHub is a web-based platform that is used to manage, store, and share software development projects. It offers a version control system and collaboration tools for developers to work together on code and other digital assets. GitHub is popular in the open-source community, but it is also used by companies to manage their proprietary code. It allows developers to easily contribute to projects, track changes, and manage project workflows. It also provides tools for issue tracking, documentation, and continuous integration and deployment.\n",
      "Time elapsed: 0.22\n",
      "GitHub is a web-based platform that is used to manage, store, and share software development projects. It offers a version control system and collaboration tools for developers to work together on code and other digital assets. GitHub is popular in the open-source community, but it is also used by companies to manage their proprietary code. It allows developers to easily contribute to projects, track changes, and manage project workflows. It also provides tools for issue tracking, documentation, and continuous integration and deployment.\n",
      "Time elapsed: 0.239\n",
      "GitHub is a web-based platform that is used to manage, store, and share software development projects. It offers a version control system and collaboration tools for developers to work together on code and other digital assets. GitHub is popular in the open-source community, but it is also used by companies to manage their proprietary code. It allows developers to easily contribute to projects, track changes, and manage project workflows. It also provides tools for issue tracking, documentation, and continuous integration and deployment.\n"
     ]
    }
   ],
   "source": [
    "for _ in range(3):\n",
    "    # use cache without temperature (temperature=0.0)\n",
    "    start = time.time()\n",
    "    response = openai.ChatCompletion.create(\n",
    "        model='gpt-3.5-turbo',\n",
    "        messages=[{\n",
    "            'role': 'user',\n",
    "            'content': question\n",
    "        }],\n",
    "    )\n",
    "    print('Time elapsed:', round(time.time() - start, 3))\n",
    "    print(response[\"choices\"][0][\"message\"][\"content\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85af6b19",
   "metadata": {},
   "source": [
    "### maximum: temperature=2.0\n",
    "\n",
    "When temperature is at its maximum value of 2, it will skip searching cache and send request to OpenAI directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "18301cae",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time elapsed: 2.675\n",
      "GitHub is a web-based platform used for version control and collaboration that helps developers store and manage their code repositories online. It allows multiple developers to work collabor\n",
      "Time elapsed: 2.667\n",
      "GitHub is a web-based platform used for version control and collaboration in software development projects. It provides a centralized location for developers to manage and store their code\n",
      "Time elapsed: 2.56\n",
      "GitHub is a web-based platform where developers can store, share, and collaborate on their code projects. It is also a version control system, meaning it\n"
     ]
    }
   ],
   "source": [
    "# use cache with temperature 2.0\n",
    "for _ in range(3):\n",
    "    start = time.time()\n",
    "    response = openai.ChatCompletion.create(\n",
    "        model='gpt-3.5-turbo',\n",
    "        temperature=2.0,\n",
    "        max_tokens=30,\n",
    "        messages=[{\n",
    "            'role': 'user',\n",
    "            'content': question\n",
    "        }],\n",
    "    )\n",
    "    print('Time elapsed:', round(time.time() - start, 3))\n",
    "    print(response[\"choices\"][0][\"message\"][\"content\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "440b120a",
   "metadata": {},
   "source": [
    "### 0.0<temperature<2.0\n",
    "\n",
    "When temperature is between 0 and 2, a higher value will increase the probability of skipping cache search and makes the output more random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "10515319",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time elapsed: 0.197\n",
      "GitHub is a web-based platform used for version control and collaboration that helps developers store and manage their code repositories online. It allows multiple developers to work collabor\n",
      "Time elapsed: 6.116\n",
      "GitHub is a web-based platform that hosts and manages software development projects using the Git version control system. It provides a collaborative environment for developers to work together on coding projects, including features such as task management, code review, and bug tracking. GitHub enables developers to share their code with the rest of the community, discover new projects and contribute to them, collaborate with others on open-source software, and showcase their work to potential employers.\n",
      "Time elapsed: 6.757\n",
      "GitHub is a web-based platform used for version control and collaboration of software development projects. It provides tools for developers to manage and store their code, as well as to collaborate with others through features such as pull requests, code reviews, and issue tracking. GitHub has become a popular platform for open-source projects and offers various features such as version control, documentation, bug tracking, task management, wikis, and more. It is widely used in the technology industry and by developers all over the world.\n"
     ]
    }
   ],
   "source": [
    "# use cache with temperature 1.0\n",
    "for _ in range(3):\n",
    "    start = time.time()\n",
    "    response = openai.ChatCompletion.create(\n",
    "        model='gpt-3.5-turbo',\n",
    "        temperature=1.0,\n",
    "        messages=[{\n",
    "            'role': 'user',\n",
    "            'content': question\n",
    "        }],\n",
    "    )\n",
    "    print('Time elapsed:', round(time.time() - start, 3))\n",
    "    print(response[\"choices\"][0][\"message\"][\"content\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da9d46f3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "develop",
   "language": "python",
   "name": "develop"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
